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ABSTRACT 

Measurement of training outcomes is a requirement 
for evaluating new training techniques, but is one that is different 
to meet. Managers of education and training may have different 
concepts of what they want, as favorable outcomes, than do the 
investigators doing the research, classical statistical and 
experimental designs assume laboratory rigor of control over 
variables that is seldom possible in the real world of a school or 
classroom. Yet in the broader perspective of educational 
institutions, the effectiveness of the instituti >ns is a current 
issue of fundamental concern in our society. In this report, 
possibilities for measuring outcomes of training are surveyed, 
considering training as a form of planned social change. Various 
approaches are discussed. Illustrations from the computer-assisted 
instruction (CAT) literature of recent attempts to measure training 
outcomes are given. The principal conclusions presented are that the 
classical four-vay design is impracticable for most evaluation 
studies in training environments; that a policy of "adaptive research 
for big effects" is apt to be scientifically and administratively 
desirable: mud that current attempts at measurement of training 
outcomes sill use fairly simple methods. (Author) 
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Measurement of training outcomes is a requirement for evaluating 
nev training techniques! but is one that Is difficult to meet. Managers 
of education and training may have different concepts of vhat they vant, 
as favorable outcomes, than do the investigators doing the research. 
Classical statistical and experimental designs assume laboratory rigor 
of control over variables that is seldom possible In the real world of 
a school or classroom. Vet In the broader perspective of educational 
Institutions, the effectiveness of these Institutions la a current 
issue of fundamental concern in our society. In this report, possi- 
bilities for measuring outcomes of training are surveyed, considering 
training as a form of planned social change. Approaches vhlch are dis- 
cussed Include the classic Solomon four-group design, Iterative adapta- 
tion to the peculiarities of Individual student progress, response sur- 
face designs, adaptive control models, decision theoty models, and sim- 
ulation models. Illustrations from the CAt literature ol recent 
attempt* to measure training outcomes are given. Ihe principal con- 
clusions presented are that the classical four-vay design Is Imprac- 
ticable for most evaluation studies in training environments; that a 
policy of "adaptive research for big effects" Is apt to be scientifical- 
ly and administratively desirable; and that current attempts at measure- 
ment of training outcomes still use fairly simple methods. 
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ABSTRACT 



Measurement of training outcomes Is a requirement for evaluating 
new training techniques, but Is one that Is difficult to meet. Managers 
of education and tralnlug may have different concepts of what they want, 
as favorable outcomes, than do the Investigators doing the researcn. 
Classical statistical and experimental designs assume laboratory rigor 
of control over variables that Is seldom possible In the real world of 
a school or classroom. Yet In the broader perspective of educational 
Institutions, the effectiveness of these Institutions Is a current 
Issue of fundamental concern In our society. In this report, possi- 
bilities for measuring outcomes of training are surveyed, considering 
training as a fora of planned social change. Approaches which are dis- 
cussed Include the classic Solomon four-group design, Iterative adapta- 
tion to the peculiarities of Individual student progress, response sur- 
face designs, adaptive control models, decision theory models, and sim- 
ulation models. Illustrations from the CAI literature of recent 
attempts to measure training outcomes are given. The principal con- 
clusions presented are that the classical four-way design Is Imprac- 
ticable for most evaluation studies In training environments; that a 
policy of "adaptive research for big effects" Is apt to be scientifical- 
ly and administratively desirable; and that current attempts at measure- 
ment of training outcomes still use fairly simple methods. 
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MEASUREMENT OF TRAINING OUTCOMES 
SECTION I. INTRODUCTION 

When someboJy says "how effective Is training program X," he can be 
asking about several aspects of X. He may want to know whether the X 
material covers the subject matter domain which Is to be taught, whether 
It does actually teach whatever It Is supposed to teach, whether It Is 
a practical program, and so forth. Ihus "effectiveness" Is apt to be a 
multidimensional concept with many ramifications. Consider one of 
Lumsdalne's tables (Table 1), which shows the kinds of Internal and 
external criteria that could be used In evaluating teaching programs, 
(Lumsdalne, 1965). 

In this report, we are mainly concerned with those external criteria 
that Lumsdalne subsumes under his "effectiveness" category that Is, 
with those Items of Information which show how well the teaching objec- 
tives are realized In the students receiving the treatment. At a few 
places we do touch upon "appropriateness" and "practicality" matters, 
there are three *reae for us to consider In this Introductory report: 

(1) the factors Involved In deciding upon specifi c performance criteria, 

(2) the selection of some compariso n design for showing effectiveness, 
and (5) some examples from the training evaluation literature. We cannot 
provide here a cookbook to solve local decisions of scoring and design; 
what we can do Is to raise some of the Issues that might give a basis 
for such decisions. It turns out that, although training effectiveness 
studies have been rather conventional so far, evaluation schemes derived 
from adaptive control and from decision theory show some promise for the 
CAI training Mnager. 
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Practicality : Esse of using Cost factors: program price 

Reusability adaptability, characteristic: 

Machine (instrumentation) of presentation, machine (if 

requirements required) 



